Working Paper 


2021-R2101 


English Learners who Take the ACT 
with Testing Supports: An 
Examination of Performance, 
Demographics, and Contextual 
Factors 


JOANN L. MOORE, PHD 


ACT.org ACT 


ACT Research & Policy | Working Paper | July 2021 1 


English Learners who Take the ACT with Testing 
Supports: An Examination of Performance, 
Demographics, and Contextual Factors 


Joann L. Moore, PhD 


Abstract 


This study examined the performance of English learners (ELs) taking the ACT® test 
with testing supports, as compared to ELs and non-ELs taking the ACT without 
supports or accommodations. Contextual factors were also explored, including high 
school experiences and demographic characteristics such as low-income and first- 
generation college student status that may be associated with their scores. Results 
indicate that ELs tended to score substantially lower than non-ELs, and ELs who 
tested with supports tended to score lower than ELs who tested without supports. 
ELs in both groups, particularly those testing with supports, tended to take fewer core 
academic courses, fewer honors or AP courses, and had lower high school grades 
than non-ELs. ELs in both groups, particularly those testing with supports, were also 
more likely to be non-White, low income, and/or first-generation college students. 
Regression analyses found that demographic and contextual factors, along with 
limited English proficiency, played a substantial role in predicting the performance of 
ELs taking the ACT, both those testing with and without supports. It is important that 
ELs are provided with a rigorous education that includes instruction in both English 
language and core academic content to ensure that they have equitable 
opportunities and experiences as compared to their English proficient peers. 


Keywords: English learners, accommodations, testing supports, standardized 
testing, college admissions testing 
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Introduction 


Approximately five million students in the US are identified as English learners (ELs), 
and the percentage of ELs has increased in recent years from 8.1% in 2000 to 10.1% in 
2017 (NCES, 2020). ELs are a diverse group of students, differing with respect to their 
native language, age or grade level at which they entered the US, levels of proficiency 
in their native language and in English, years of instruction and academic content 
learned in their native language and in English, and many other characteristics. 
English proficiency is also a moving target, such that students should be improving 
and eventually become English proficient, assuming that they are receiving 
appropriate instruction. These factors all contribute to difficulties in studying or 
making generalized conclusions about ELs as they are an immensely heterogeneous 
population. 


ELs may receive various levels of modified instruction in which they are removed 
from mainstream classrooms for English instruction or are placed into less rigorous 
academic courses, limiting their opportunity to learn challenging content relative to 
their English-proficient peers (Callahan, 2005). ELs tend to be exposed to fewer core 
academic courses in high school, graduate at lower rates, and enroll in college at 
lower rates than their English-proficient peers (Callahan & Shifrer, 2016; Johnson, 
2019; Sugarman, 2019). ELs also tend to perform at a much lower level on 
standardized tests, in part due to lower proficiency in English, which can impact their 
ability to demonstrate their true achievement level (Abedi, 2002; Sugarman, 2019). It 
is difficult to determine how much of ELs’ lower performance is due to language 
difficulties and how much may be due to lower levels of academic proficiency, but it 
is likely that both are contributing factors (Callahan, 2005). 


Historically, ELs were routinely excluded from standardized testing (Abedi, 2014; 
Heilig & Darling-Hammond, 2008) and from samples used to create norms for 
standardized tests (Abedi, 2002). However, beginning with the No Child Left Behind 
Act (NCLB; Public Law PL 107-110, the No Child Left Behind Act of 2001), the US 
Department of Education began requiring that ELs be included in standardized 
testing and that their performance be reported as a disaggregated subgroup, 
bringing attention to their lower performance and substantial achievement gaps 
compared to their English-proficient peers. Further, the reauthorization of NCLB as 
the Every Student Succeeds Act (ESSA) allows districts to use a “nationally recognized 
high school academic assessment” such as the ACT or SAT in lieu of a state 
assessment for accountability, requiring that English learners have an opportunity to 
participate in and benefit from the assessment (US Department of Education, 2016). 


The ACT is a college readiness assessment used primarily for college admissions, 
course placement, scholarship eligibility, statewide assessment and accountability, 
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and to provide information to students about their relative strengths and areas for 
improvement (ACT, 2020). The ACT is comprised of four multiple-choice section tests: 
English, math, reading, and science, with an optional writing test. The four section 
tests are scaled from 1-36, and a Composite score, based on the average of the four 
section tests, is also reported on a 1-36 scale. Based on a national sample of data from 
2018-2020 high school graduates, average ACT scores were 20.7 for the Composite, 
20.1 in English, 20.4 in math, 21.2 in reading, and 20.6 in science (ACT, 2021). 


Most students in the US take the ACT as part of either National or State and District 
testing. National testing refers to the traditional testing model in which students 
register to take the ACT at a test center and typically test on a Saturday. State and 
District testing refers to a testing model in which school districts or states provide in- 
school ACT testing, typically to all students in 11th grade in that state or district (ACT, 
2019a). Because students choose to test on a national test date, these students are 
more likely to be higher achieving, college-bound students, whereas students testing 
as part of State and District testing include some students who otherwise would not 
have taken the ACT. As a result, average ACT scores for State and District testing tend 
to be slightly lower than average scores for National testing (Allen, 2015). 


In the fall of 2017, ACT began providing testing supports (or accommodations) to 
eligible ELs in the US taking the ACT, including translated test instructions, word-to- 
word bilingual dictionaries (with no definitions), small group testing, and extra time. 
The goal of these supports is to remove construct-irrelevant variance and allow 
students with limited English proficiency to more accurately demonstrate their true 
achievement level. Students can request the supports when they register to take the 
ACT. ELs are eligible for testing supports if they attend school in the United States, US 
territories, or Puerto Rico and meet the criteria outlined in ESSA for being an EL, 
including limited English proficiency and receiving EL services in school. Once the 
request is made, students work with school officials to provide relevant 
documentation to establish their EL status and ensure that they meet the 
requirements for use of the supports. ACT reviews the request and decides whether 
the supports are approved for use when the students take the test. It is unknown 
what motivates some EL students to request the supports or why some EL students 
do not take advantage of that opportunity. 


The purpose of this study is to examine the performance of ELs taking the ACT with 
testing supports as compared to ELs and non-ELs testing without supports and to 
gain a deeper understanding of the demographic and contextual factors that may 
influence their scores. Relationships between ACT scores and high school grades are 
also of interest since both are measures of academic performance, and it is expected 
that students with higher grades will earn higher ACT scores. 
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ELs face many barriers in addition to the challenges of learning both English and 
academic content, and it is important that they receive appropriate instruction and 
support in school so that they can become English proficient and become college 
and career ready by the time they graduate high school. A study of ELs in Ist through 
9th grade found that it typically takes four to seven years for ELs to attain academic 
English proficiency (Hakuta, Butler, & Witt, 2000), which has implications for whether 
an EL has sufficient time to become English proficient by graduation, depending on 
their level of English proficiency when they enter high school. Most students take the 
ACT in 11th or 12th grade, at which point they are typically taking coursework that 
requires advanced subject-specific academic vocabulary. If ELs do not have sufficient 
academic English proficiency, they may struggle to learn academic content despite 
being exposed to it in the classroom (Herman & Abedi, 2004). 


Research Questions 


1. How does the performance of ELs testing with or without supports on the ACT 
compare to the performance of non-ELs? 


2. How do ACT-tested ELs testing with or without supports compare 
demographically to ACT-tested non-ELs? 


3. How do high school experiences of ACT-tested ELs testing with or without 
supports compare to non-ELs? 


4. What is the relationship between high school grade point average (HSGPA) 
and ACT performance for ACT-tested ELs testing with or without supports 
compared to ACT-tested non-ELs? 


5. How much do demographics, HSGPA, and high school experiences account for 
differences in the average ACT scores of ELs testing with or without supports 
relative to non-ELs? 


Sample 


The sample for this study includes students who took the ACT between September 
2017 (when ACT first began offering supports to ELs) and July 2019 and received 
college-reportable scores.' Students’ ACT test records were matched to ACT’s Test 
Accessibility and Accommodations (TAA) System using student name, date of birth, 
ACT ID, and residential address. Students were identified as EL if they (a) indicated 
during test registration that they currently receive EL services at school, and/or (b) 
were found to have received EL testing supports in the TAA system. 


Students were removed from the study sample if they did not test in the US, US 
territories, or Puerto Rico (2% of the sample); if they did not complete the full test and 
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receive a Composite score (less than 1%); or if they tested as part of State and District 
testing for which there were contract or state law restrictions on data use (15%). 
Students were also removed from the study sample if their accommodations were 
due to disabilities instead of or in addition to testing supports due to EL status (4%) or 
if EL status could not be determined because the student did not indicate their EL 
status when registering for the ACT (18%).? 


For students who took the ACT more than once within the testing window and met 
the inclusion criteria (51% of the study sample), their first test event was retained to 
minimize the influence of practice effects. Students whose first test event was prior 
to the testing window (25%) were also removed from the sample to minimize 
practice effects. Table 1 contains the distribution of test groups in the study sample. 
Test group was defined as “EL with Supports” for students who had been approved to 
take the ACT with one or more testing supports for ELs, “EL, No Supports” for students 
who reported that they received EL services in school at the time they registered to 
take the ACT but did not receive testing supports or accommodations, and “Non-EL, 
No Supports’ for students who reported that they did not receive EL services in school 
at the time they registered and did not receive testing supports or accommodations. 


Table 1. Counts and Percentages of Students in Each Test Group 


Test Group Count Percent | 


EL with Supports 15,137 0.7 

EL, No Supports 214,252 9.5 

Non-EL, No Supports 2,022,714 89.8 

Total 2,252,103 100.0 
Methods 


Descriptive analyses including means and percentages were conducted to 
understand the characteristics of the sample (Research Questions 1-3). Correlation 
analyses were conducted to evaluate the relationship between HSGPA and ACT 
performance for each subject area and the Composite score (Research Question 4). A 
series of regression analyses were conducted to evaluate relationships amongst 
demographics, HSGPA, high school experiences, and ACT performance (Research 
Question 5). A separate set of models were estimated for each subject area and the 
Composite score. For each set of models, the first model only included test group, a 
second model added demographics, a third model added high school experiences 
and aspirations, and a fourth model added HSGPA. Several approaches were 
considered to account for missing data and ensure robustness of the results, 
including complete cases and multiple imputation. 
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Analyses were not disaggregated by specific test supports due to the majority of ELs 
testing with more than one support. Most ELs who tested with supports were 
approved for two or more supports (88%), and more than half (62%) were approved 
for three or four supports. The most common support was extra time; 99% of ELs who 
tested with supports were approved for extra time, and 80% of students were 
approved to use a word to word bilingual dictionary. Additional details about the 
numbers of supports and types of supports are presented in the Appendix, Table Al. 


Results 


Research Question 1: How does the performance of ELs testing 
with or without supports on the ACT compare to the 
performance of non-ELs? 


Table 2 contains average ACT scores by test group. ELs who tested with supports had 
the lowest performance, and non-ELs who tested without supports had the highest 
performance. ELs who tested without supports likely had higher levels of English 
proficiency than those who tested with supports; while the ACT English test was not 
explicitly designed as an English language proficiency test, the lower English scores 
of ELs who tested with supports provides evidence that this is the case. Both EL test 
groups tended to have higher scores in math and science relative to English and 
reading, whereas non-ELs tended to have higher scores in reading. This is consistent 
with the literature (Abedi, 2001; Abedi, Leon, & Mirocha, 2003). Note that the scores 
cannot be directly compared across subject areas (i.e.,a 15 in math and a 16 in reading 
does not mean that the student is demonstrating greater proficiency in reading 
compared to math), but we can compare relative performance across the test groups 
to understand their relative academic strengths and challenges. Differences in scores 
of ELs who tested with supports compared to non-ELs were largest in English (8.6) 
and reading (8.1) and smallest in math (5.3). Comparisons between ELs who tested 
without supports and non-ELs showed a similar pattern, but differences were smaller 
in magnitude. ELs who tested with supports also had lower standard deviations of 
scores, followed by ELs who tested without supports; non-ELs had the highest 
standard deviations of scores. 


Table 2. Average ACT Scores and Standard Deviations (SD) by Test Group 


EL with Supports = CoE] 0) of oy a Coy a =| Mm CofE] 0) ofo) a iy 
Subject Area Mean A] D) Mean KJ D) Mean KJ D) 
English 12.4 3.6 16.1 5.4 21.0 6.5 
Math 15.8 3.6 17.7 4.3 21.1 5.4 
Reading 143 4.0 17.5 5.4 22.3 6.4 
Science 15.0 3.9 17.8 4.6 21.5 53 


Composite 14.5 3.1 17.4 4.4 21.6 5.3 
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To obtain a more complete representation of how students were performing beyond 
simple means and standard deviations, Figure 1 contains the distributions of ACT 
Composite scores by test group. Score distributions of ELs who tested with supports 
were highly skewed (fi, = 2.2) and centered around 12-14; score distributions of ELs 
who tested without supports were less skewed (ji; = 1.0) and centered around 14-16. 


Score distributions for non-ELs were more symmetrically distributed (ji, = 0.4) and 
centered around 19-21. 


Figure 1. Distribution of ACT Composite Scores by Test Group 


25 
20 


15 
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10 


] 6 1 16 21 26 31 36 
ACT Composite 


—— EL with Supports = EL, No Supports -===Non-EL, No Supports 


Table 3 contains the percentages of students meeting each of the ACT College and 
Career Readiness Benchmarks by test group. The Benchmarks are the scores at which 
a student has a 50% chance of earning a B or higher in a first-year, credit-bearing 
course at a typical college. As would be expected given the average ACT scores of 
each group (Table 2), ELs who tested with supports were much less likely to meet the 
Benchmark for each subject, followed by ELs without supports, then the Non-EL 
group. Across groups, the English Benchmark was the most likely to be met, and the 
Science Benchmark was the least likely to be met, which is consistent with previous 
research of national samples of ACT-tested high school graduates (ACT, 2019b). 
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Table 3. Percent of Students Meeting ACT Benchmarks by Test Group 


‘ ACT EL with Non-EL, No 
Suprect Slee Benchmark RJ] o) ofey any ED No uRports KJV] o} ole) any 
English 18 8 34 67 
Math 22 7 19 44 
Reading 22 6 21 52 
Science 23 5 15 4] 


Research Question 2: How do ACT-tested ELs testing with or 
without supports compare demographically to ACT-tested non- 
ELs? 


When students register to take the ACT, they are asked to provide information about 
themselves on a variety of topics, including race/ethnicity, gender, parent education 
and income, educational plans and aspirations, high school experiences and 
accomplishments, and high school coursework (courses taken, courses planned, and 
grades earned in courses taken). 


Table 4 contains the percentages of students who did not provide information for the 
categories of registration information that are of interest in this paper, by test group. 
ELs who tested with supports were much less likely to have completed the 
registration information; this is important to keep in mind when interpreting the 
findings of this report as the students who completed the registration information 
may have systematically differed from those who did not complete the information.* 
Note that some questions also allowed for “I prefer not to respond” options; the 
percentages presented in Table 4 only include students who did not respond to the 
iterm(s) at all. 


Table 4. Percentages of Missing Registration Category Responses by Test Group 


. : EL with LN oY a bo =| Me [o) 
Registration Category copecne = Co TUT 0) of oy a Supports 
Race/Ethnicity 10 2 1 
Parent Income 67 22 23 
Parent Education 56 8 7 
Coursework/Grades 55 15 14 
Semesters of Coursework 30 2 ] 
Honors/AP 79 40 32 
Extracurricular Activities 72 22 18 


Degree Aspirations 56 8 6 
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Table 5 contains testing and demographic information by test group. The 
percentages of students testing during National or State and District testing, by test 
group, help contextualize the findings in this report. ELs who tested with supports 
were more likely to have tested during State and District testing (54% State and 
District vs. 46% National) whereas non-ELs were much more likely to have tested 
during National testing (83% National vs. 17% State and District). This is due in part to 
how the test groups were defined. Students were more likely to provide a response to 
the EL status question when testing as part of National testing (91%) than as part of 
State and District testing (49%), meaning that more students who tested as part of 
State and District testing were removed from the study sample because their EL 
status could not be determined. It is also likely that more ELs tested during State and 
District testing because all students were tested, including those who may not have 
been planning to attend college. 


Table 5. Demographic Characteristics by Test Group (Percentages) 


EL with |= BN fo) Non-EL, No 
RLU] 0) ofo) a 3 LUT 0) ofo) a 3 RLU] 0) ofo) a 3 

Test Type 

National 46 60 83 

State and District 54 40 17 
Grade Level Tested 

Tith Grade 71 65 62 

12th Grade 26 26 27 

Other/Missing 3 9 12 
Race/Ethnicity 

Black/African American 15 22. 12 

American Indian/Alaska Native oO 2 1 

White 5 44 60 

Hispanic/Latino 60 22 17 

Asian 19 5 6 

Native Hawaiian/Pacific Islander 1 1 oO 

Two or More Races O & 5 
Income 

Less than $24,000 47 26 12 

$24,000-$36,000 25 18 11 

$36,000-$50,000 1 12 11 


$50,000-$60,000 6 
$60,000-$80,000 4 
$80,000-$100,000 2 8 12 
$100,000-$120,000 2: 
$120,000-$150,000 1 
More than $150,000 2: 
First-Generation College Student 58 34 18 
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Because large differences were found in the percentages of missing registration data 
by test group (Table 4) and in the percentages of students testing in a National versus 
State and District context by test group (Table 5), the prevalence of missing data in 
the student registration information was compared for the three test groups by 
National versus State and District testing conditions (see Appendix Table A2). While 
there were some differences in the patterns of missing data (i.e., ELs who tested with 
supports were somewhat more likely to omit information during State and District 
testing, and ELs who tested without supports and non-ELs were somewhat more 
likely to omit information during National testing), in general, ELs who tested with 
supports had more missing data regardless of testing context. It was hypothesized 
that the larger percentages of missing information for ELs who tested with supports 
could have been due to educators in the State and District testing context choosing 
not to have ELs testing with supports complete the registration information, but 
because this pattern was found across testing contexts, the data suggests that this 
did not play a large role in the response rate discrepancies. 


Table 5 also contains the percentages of students who tested in 11th grade, 12th 
grade, or other, by test group. Most students tested in 11th or 12th grade, whereas a 
small percentage of students tested in earlier grades or after high school graduation. 
ELs who tested with supports were somewhat more likely to have tested in 11th grade 
(71%) compared to ELs who tested without supports (65%) or non-ELs (62%). These 
results are unsurprising given that higher percentages of ELs who tested with 
supports did so through State and District testing. 


Demographic characteristics provide important context for analyses comparing ELs 
to non-ELs, as it is well known that factors such as race/ethnicity and income are 
related to availability and quality of educational experiences and opportunities and 
therefore are related to academic performance as measured by test scores. Table 5 
contains the self-reported race/ethnicity of students by test group. Larger proportions 
of Hispanic/Latino and Asian students were in the ELs who tested with supports 
group, whereas larger proportions of White students were non-ELs. 


Table 5 also contains parent income by test group, defined as the combined income 
of their parents before taxes. Again, these percentages were calculated based on non- 
missing responses; the missing rates for this question was 67% for ELs who tested 
with supports, 22% for ELs who tested without supports, and 23% of non-ELs. Both EL 
groups tended to report lower income than non-ELs, and ELs who tested with 
supports tended to report the lowest income, with 73% reporting parent income of 
$36,000 or less, compared to 44% of ELs who tested without supports and 23% of 
non-ELs. 


Lastly, Table 5 contains the percentages of first-generation college students by test 
group, based on non-missing responses. First-generation college students are defined 
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as neither parent having any college experience (i.e. non-degree completion would 
count as “some college experience”). ELs were more likely to be first generation 
college students than non-ELs, particularly ELs who tested with supports, who were 
3.2 times more likely to be first generation college students than non-ELs. 


Research Question 3: How do high school experiences of ACT- 
tested ELs testing with or without supports compare to ACT- 
tested non-ELs? 


ELs’ high school experiences and access to educational opportunities provide 
important context for interpreting their performance on the ACT. ACT has long 
advocated that students take a core curriculum, defined as a minimum of four years 
of English and three years each of math, science, and social studies. Research has 
shown that students who take a core curriculum earn higher ACT scores, enroll in 
college at higher rates, earn higher grades in college, and are less likely to need 
remedial courses in college than students who take less than the core (ACT, 2006). 
Based on information about high school coursework taken or planned provided by 
students at registration, ELs who tested with supports were much less likely to report 
taking or planning to take a core curriculum by the time they graduate high school 
(52%) compared to ELs who tested without supports (67%) and non-ELs (78%). 


Table 6 contains the percentages of students who reported having taken or planning 
to take specific high school courses by test group (conditioned on providing a 
response to account for the disproportionate higher non-response rates of ELs who 
tested with supports). With a small number of exceptions, ELs who tested with 
supports were less likely to have taken or planned to take many of the courses than 
students in the other test groups. This is likely due in part to ELs taking English 
language courses during one or more class periods each day, which limits their 
access to other courses (Callahan & Shifrer, 2016; Johnson, 2019). Unfortunately, some 
of the largest gaps in course taking between ELs who tested with supports and non- 
ELs who tested without supports were in core academic courses including 
trigonometry (12% point gap, meaning the percentage of ELs who tested with 
supports took trigonometry was 12 percentage points lower than non-ELs), calculus 
(13% point gap), other advanced math (18% point gap), and chemistry (15% point 
gap). 
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Table 6. Percentages of Students Taking or Planning to Take High School Courses by Test 
Group 


: EL with 3 Co) Non-EL, No 
lke Lawexedalere) MOCol UL ex=) 
RLU] 0) ofo) a 3 AUT 0) ofo) a 3 Supports 

English, Grade 9 92 99 100 
English, Grade 10 95 99 100 
English, Grade 11 97 99 99 
English, Grade 12 94 97 98 
Other English 35 29 24 
Algebra 1 95 98 99 
Algebra 2 87 95 98 
Geometry 92 96 98 
Trigonometry 40 47 53 
Calculus 39 45 52 
Other Advanced Math 51 63 70 
Computer Science 31 26 25 
General Science 85 87 83 
Biology 97 99 99 
Chemistry 77 87 92 
Physics 60 63 66 
US History 97 99 99 
World History 90 94 94 
Other History 44 38 38 
American Government 85 86 88 
Economics 69 65 67 
Geography 51 55 49 
Psychology 37 44 47 
Spanish 68 78 76 
French 28 19 16 
German 16 8 6 
Other Language 32 18 16 
Art 74 66 60 
Music 43 44 43 
Drama 28 25 20 


Interestingly, there were a few courses that ELs who tested with supports were more 
likely to have reported taking or planning to take, including “other English courses,” 
where ELs could expect to improve their English proficiency. Computer science, other 
history, economics, French, German, other language, art, and drama also had higher 
percentages of ELs who tested with supports taking these courses compared to the 
other two test groups. 
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Table 7 contains the average number of years of coursework students expected to 
have taken by the time they graduate high school by subject. For the four core 
subjects (English, math, social sciences, and natural sciences), ELs in both groups 
anticipated taking fewer years of coursework than non-ELs, and ELs who tested with 
supports anticipated taking the fewest years of core coursework. ELs who tested with 
supports were slightly more likely to anticipate taking coursework in a language 
other than Spanish, German, or French as compared to the other two test groups. 
While not available in the data, it would be interesting to know whether the language 
taken was their native language or another language. 


Table 7. Average Years of Coursework by Test Group 


A LS Co) ab =| Ba Co) 
EL with Supports EL, No Supports Seppo 

Subject Area 

English 3.0 1.2 3.2 1.0 3.3 0.9 
Math 3.0 1.1 3.2 1.0 3.4 0.8 
Social Sciences 2.6 1.2 2.9 1.0 3.1 0.9 
Natural Sciences 2.6 12 2.9 1.0 3.1 0.9 
Spanish 3 1.4 1.4 1.2 1.7 9 9 
German 0.2 0.6 0.1 0.5 0.1 0.5 
French 0.3 0.9 0.3 0.8 0.3 0.9 
Other Language 0.5 1.2 0.3 0.8 0.3 0.9 


Table 8 contains the percentages of students who reported participating in 
Advanced Placement (AP), accelerated, or honors courses by subject and test group. 
Again, these percentages are based on non-missing responses; large percentages of 
students did not complete these sections, particularly ELs who tested with supports 
(Table 4). ELs in both groups, particularly ELs who tested with supports, were much 
less likely to have taken AP/honors courses, with the exception of foreign languages. 
The largest gaps in AP/honors course taking between ELs who tested with supports 
and non-ELs were in English (41%) and reading (35%). 
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Table 8. Participation in AP/Honors Courses and Extracurricular Activities by Test Group 


EL with a Co) Non-EL, No 
Supports TUT 0) ofo) a 3 Supports 
AP/Honors Subject 
English 30 52 7I 
Math 39 45 62 
Social Sciences 29 45 64 
Natural Sciences 31 43 62 
Foreign Language 33 24 29 
Extracurricular Activity 

Instrumental Music 20 25 26 
Vocal Music 17 19 17 
Student Government 16 15 16 
Publications 1 13 12 
Debate 10 9 8 

Dramatics/Theater 15 18 18 
Religious Organizations 12 18 24 
Racial/Ethnic Organizations 13 6 6 

Varsity Athletics 35 54 57 
Political Organizations 6 5 4 

Radio/TV Z 5 3 

Social Clubs 16 15 12 
Community Service 38 45 57 


Table 8 also contains percentages of students reporting participation in high school 
extracurricular activities by test group. In general, there were not large discrepancies 
in participation by test group, with a few exceptions. ELs who tested with supports 
were less likely to participate in religious organizations, varsity athletics, or 
community service, and more likely to participate in racial or ethnic organizations. 


Table 9 contains the educational aspirations of students by test group. Educational 
aspirations are defined as the highest level of education that students expect to 
complete. ELs in both groups had only slightly lower degree aspirations as compared 
to non-ELs, with roughly half of students wanting to earn a bachelor’s degree and 
another fifth wanting to earn a doctorate or professional degree. Again, these 
percentages are based on non-missing responses; 56% of responses of ELs who tested 
with supports were missing degree aspiration information compared to 8% and 6% 
for ELs testing without supports and non-ELs, respectively. 
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Table 9. Distribution of Educational Aspirations by Test Group 


F wack EL with = To) Non-EL, No 
Educational Aspirations 
LUT 0) ofo) a 3 LUT 0) ofo) a 3 RLU] 0) ofo) a 3 

Business/Tech/Certificate 7 4 1 
Associate Degree 13 10 4 
Bachelor’s Degree 46 53 49 

1-2 Years Graduate Study 8 9 20 
Doctorate/Professional Degree 19 19 25 
Other 8 4 ] 


Research Question 4: What is the relationship between high 
school grades (HSGPA) and ACT performance for ACT-tested ELs 
testing with or without supports compared to non-ELs? 


Table 10 contains the percentages of students reporting HSGPA by test group and 
test type. Students report this information at the time they register for the ACT by 
selecting from a list of courses, indicating whether each course was taken, planned, 
or not planned, and providing grades earned (A, B, C, D, or F) for courses taken. An 
unweighted GPA ona 4.0 scale is calculated based on these responses; this is the 
HSGPA reported in Table 10. While there may be concerns that self-report may not be 
as accurate as official transcripts, previous research has found that self-reported 
grades on the ACT are highly accurate (Sanchez & Buddin, 2015). Overall, ELs who 
tested with supports were less likely to report grades (45%), whereas ELs who tested 
without supports and non-ELs reported grades at similar rates (85% and 86%, 
respectively). ELs who tested with supports as part of State and District testing were 
the least likely to report their grades (33%). 


Table 10. Number and Percent of Students Reporting HS Grades by Test Group and Test Type 
EL with Co) Non-EL, No 


Overall 
RLU] 0) ofo) a RUT 0) fo) a 3 Supports 
National 4,117 (59%) 111,596 (86%) 1,440,814 (86%) 1,556,527 (86%) 
State and District 2,691 (33%) 69,676 (82%) 297,715 (86%) 370,082 (84%) 
1,926,609 
Total 6,808 (45%) 181,272 (85%) 1,738,529 (86%) (86%) 
(e} 


Table 11 contains the average ACT Composite scores of students by test group and 
whether they reported their grades. Across the three groups, students who reported 
their grades tended to be somewhat higher performing than those who did not 
report their grades. 
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Table 11. Average ACT Composite of Students by Test Group and Reporting of Grades 


EL with Co) Non-EL, No 
ALU] 0) ofo) a 3 TUT 0) ofo) a 3 Supports 
Reported Grades 15.1 17.6 21.8 
Did Not Report Grades 14.0 16.3 20.8 


Table 12 contains students’ average self-reported HSGPA by test group and subject 
area. Consistent with overall ACT performance, ELs who tested with supports 
reported the lowest HSGPA, on average, and non-ELs had the highest HSGPA. In 
contrast to ACT section test scores (Table 2), where ELs in both groups scored 
substantially lower in English relative to the other subject areas, students in all three 
test groups tended to have higher average HSGPA in English and social studies 
courses. In contrast to the variability of students’ average ACT scores (Table 2), ELs 
who tested with supports had higher variability (SD) of HSGPA compared to the other 
test groups, followed by ELs who tested without supports; non-ELs had the lowest 
variability of HSGPA. 


Table 12. Average Self-Reported HSGPA by Test Group and Subject Area 


5 I Co) at =| ey Co) 
EL with Supports = CoE] 0) of oy a Scape 

Subject Area 

English 3.04 0.82 3.18 0.74 3.49 0.62 
Math 2.92 0.93 3.03 0.83 3.34 0.70 
Social Sciences 3.04 0.82 3.27 0.74 3.54 0.59 
Science 2.95 0.86 3.12 0.77 3.42 0.64 
Overall 2.99 0.71 3.15 0.65 3.44 0.55 


Figure 2 displays the distribution of self-reported HSGPA by test group. Non-ELs were 
much more likely to earn a HSGPA of 3.8-4.0, whereas ELs in both groups were more 
likely to earn a GPA in the 2.0-3.0 range. In fact, 19% of non-ELs reported a HSGPA of 
4.0, compared to 9% of ELs who tested without supports and 8% of ELs who tested 
with supports. Of note is the negative skew of the distribution of HSGPA relative to 
the distribution of ACT scores seen in Figure 1, which has been found in other 
research (ACT, 2012). Contrary to the results for the ACT Composite, the skew for 
HSGPA was more pronounced for non-ELs (j/3 = -1.2) as compared to ELs who tested 
with (fiz = -0.6) or without (f/3 = -0.8) supports. 
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Figure 2. Distribution of HSGPA by Test Group 
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Table 13 contains correlations between ACT scores and HSGPA by test group. 
Correlations between subject-area specific HSGPA and overall HSGPA were high for 
all three test groups (ranging from r = 0.83 to 0.87); therefore, only overall HSGPA is 
presented.° Correlations between ACT scores and HSGPA were lowest for ELs who 
tested with supports. This is likely due to differences in the ACT score distributions 
across test groups; ELs who tested with supports had the smallest variability in scores, 
followed by ELs who tested without supports, and non-ELs had the largest variability 
in scores. Despite these differences, moderately-sized correlations were still found for 
ELs who tested with supports. While the differences in score distributions make 
comparisons across test groups problematic, comparisons within test group across 
subject area can provide information about the relative strength of relationships 
between HSGPA and ACT section test scores. Correlations between ACT scores and 
HSGPA were more variable across subject for ELs who tested with supports, ranging 
from 0.27 (reading) to 0.35 (math), whereas correlations for the other two test groups 
were more similar across subject areas, ranging from 0.43 (reading) to 0.46 (English 
and math) for ELs who tested without supports and ranging from 0.46 (reading) to 
0.51 (math) for non-ELs. 
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Table 13. Correlations between ACT Subject Scores and Overall HSGPA by Test Group 


7 Xog WIV] 0} (-toa Ug -¥- | 


English Math =t-¥-To [fale | Science Composite 
EL with Supports 0.31 0.35 0.27 0.31 0.37 
EL, No Supports 0.46 0.46 0.43 0.45 0.50 
Non-EL, No Supports 0.50 0.51 0.46 0.49 0.54 


Note. All correlations were significant at p < 0.0001. 


Research Question 5: How do demographics, HSGPA, and high 
school experiences of ACT-tested ELs relate to their performance 
on the ACT? 


Due to the large amount of missing data in the covariates, particularly for ELs who 
tested with supports, multiple approaches were considered to answer this research 
question. It is likely that the data were not missing at random, and there could be 
different underlying mechanisms for missing data across the three groups. 
Regression analyses using only complete cases were compared to a multiple 
imputation approach, and results were highly similar; regression estimates were 
largely in the same direction and statistically significant, and the same conclusions 
would be drawn regardless of model used. Therefore, results are presented for the 
complete cases approach, and the possibility of bias in the estimates should be noted 
as a limitation when interpreting the findings. 


Table 14 contains the regression results for four models predicting ACT Composite 
score. The first model only includes dummy variables for the test groups; the second 
model adds demographics variables (race/ethnicity, low income, and first-generation 
college student indicator); the third model adds students’ high school experiences 
(semesters of coursework taken by subject area and honors/AP coursework taken by 
subject area)’ and college aspirations; and the fourth model adds students’ self- 
reported HSGPA. R-square estimates from each model are also reported. The 
reference groups are non-EL for test group, White for race/ethnicity, and less than a 
bachelor’s degree for college aspirations. The sample size was 2,252,103 for the first 
and second models, 1,147,011 (51% of the total sample) for the third model, and 
1,051,504 (47% of the total sample) for the fourth model. Models were also examined 
using only the subset of students included in the fourth model; minimal differences 
in the beta estimates were found, providing further evidence that the findings were 
robust to missing data. 


As summarized in Table 14, ACT Composite scores were strongly related to test group 
(EL status), demographics, experiences, aspirations, and HSGPA. Based on EL status 
alone, we would expect average ACT Composite scores to be 7.13 points lower for 
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students who tested with supports and 4.20 points lower for ELs who tested without 
supports, as compared to non-ELs who tested without supports. Taking into account 
demographics, the predicted differences in Composite scores dropped to 6.3 and 3.02 
points—a reduction of 12% and 28%, respectively. Once high school experiences, 
aspirations, and HSGPA were taken into account, the predicted differences were 3.09 
and 1.83 points—a reduction of 57% and 56%, respectively. Accounting for student 
experiences and HSGPA also reduced the magnitude of the impact of demographics 
on predicted Composite score, as can be seen by comparing beta estimates across 
the models. 


Similar results were found when comparing models that predict specific subject area 
scores using the same sets of covariates, presented in Tables 15 through 18. 
Demographics, experiences, aspirations, and HSGPA accounted for smaller percent 
changes in predicted scores in English (49% and 56%) and reading (54% and 52%), 
respectively, for ELs who tested with and without supports. This finding is not 
surprising given that English proficiency would have a larger impact on these subject 
areas. The largest reduction in predicted scores was found in math, where the full 
model reduced the beta estimates by 72% and 61%, respectively, for ELs who tested 
with and without supports. In other words, when taking into account student 
demographics, experiences, and HSGPA, ELs who tested with supports would only be 
expected to score 1.52 points lower in math than non-ELs (vs. 5.34 without accounting 
for the covariates), and ELs who tested without supports would only be expected to 
score 1.33 point lower in math than non-ELs (vs. 3.41 when not accounting for the 
covariates). Interestingly, semesters taken of math coursework was more highly 
related to higher scores across all subject areas than semesters of coursework taken 
in other subject areas (i.e., an additional semester taken of math predicted higher 
English scores than an additional semester taken of English). 


Figure 3 summarizes the unadjusted (Model 1) and adjusted (Model 4) mean 
differences in ACT scores. Comparing non-ELs to ELs with supports, unadjusted mean 
differences ranged from 5.3 (math) to 8.6 (English), whereas adjusted mean 
differences ranged from 1.5 (math) to 4.4 (English). Comparing non-ELs to ELs without 
supports, unadjusted mean differences ranged from 3.4 (math) to 4.9 (English), 
whereas adjusted mean differences ranged from 1.3 (math) to 2.3 (reading). 
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Table 14. Regression Models Predicting ACT Composite Score 


E Model 1 Model 2 Model 3 Model 4 
WETGE) o) (=) 
Beta SE Beta SE |= T=) #-) SE Beta SE 

Intercept 21.63 0.00 22.87 0.00 16.40 0.07 7.38 0.07 
EL, with Supports 27S: 0.04 -6.30 0.04 -o:27 0.13 -3.09 0.13 
EL, no Supports -4.20 0.01 -3.02 0.01 -1.94 0.02 -1.83 0.02 
Black -3.83 0.01 -3.59 0.01 “2:71 0.01 
American Indian -3.52 0.04 -2.70 0.06 -2.26 0.06 
Hispanic -1.56 0.01 -1.64 0.01 -1.20 0.01 
Asian 3.00 0.01 1.43 0.01 148 0.01 
Pacific Islander -2.20 0.06 -1.41 0.09 -114 0.09 
Multiple Race/Ethnicities -0.41 0.02 -0.29 0.02 -0.03"5 0.02 
Missing Race/Ethnicity -0.16 0.02 0.19 0.02 0.32 0.02 
Low Income (< $36,000) -1.81 0.01 -1.30 0.01 “115 0.01 
Income Missing 0.38 0.01 0.59 0.01 0.52 0.01 
First Generation College Student -2.37 0.01 -1.58 0.01 -1.41 0.01 
Parent Education Missing -0.80 0.01 0.10 0.02 0.06** 0.02 
Semesters HS English -0.31 0.01 -0.32 0.01 
Semesters HS Math 1.50 0.01 1.19 0.01 
Semesters HS Science 0.14 0.01 0.05 0.01 
Semesters HS Social Studies -0.59 0.01 -0.60 0.01 
Semesters HS Foreign Language -0.21 0.01 -0.11 0.01 
Semesters HS Arts -0.01"S 0.01 -0.02 0.01 
Honors/AP English 0.69 0.01 0.47 0.01 
Honors/AP Math 1.45 0.01 1.00 0.01 
Honors/AP Social Studies 0.95 0.01 0.96 0.01 
Honors/AP Science 0.92 0.01 0.75 0.01 
Honors/AP Foreign Language 0.91 0.01 0.87 0.01 
Aspirations: Bachelor's 1.24 0.04 0.75 0.04 
Aspirations: 1-2 Years Graduate Study 2.65 0.04 1.92 0.04 
Aspirations: Doctorate 2.50 0.04 1.63 0.04 
Aspirations: Other 1.01 0.06 0.67 0.06 
Aspirations: Missing 277 0.05 2:15 0.05 
HSGPA 3.22 0.01 
R-Square 0.06 0.24 0.36 0.41 


Note. All estimates are significant at p < 0.0001 except where ns (p > 0.05) or asterisked. * p < 0.05. ** p < 0.01. *** p < 0.001. 
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Table 15. Regression Models Predicting ACT English Score 


Ee Model 1 Model 2 Model 3 Model 4 
WETGE) o) (=) 
Beta SE Beta SE Beta SE Beta SE 

Intercept 21.03 0.00 22.43 0.01 14.83 0.08 4.34 0.10 
EL, with Supports -8.58 0.05 -7.63 0.05 -4.57 0.16 4.35 0.17 
EL, no Supports -4.90 0.01 -3.54 0.01 -2.30 0.02 -2.17 0.02 
Black -4.19 0.01 -3.86 0.02 -2.83 0.02 
American Indian -4.30 0.04 -3.39 0.07 -2.88 0.07 
Hispanic -1.85 0.01 -1.93 0.01 1.42 0.02 
Asian 3.19 0.02 1.61 0.02 1.67 0.02 
Pacific Islander -2.43 0.08 -1.44 0.11 “AN 0.11 
Multiple Race/Ethnicities -0.43 0.02 -0.30 0.02 0.01"S 0.02 
Missing Race/Ethnicity -0.09 0.02 0.35 0.03 0.52 0.03 
Low Income (< $36,000) -2.02 0.01 -1.54 0.02 -1.34 0.02 
Income Missing 0.56 0.01 0.83 0.01 0.74 0.01 
First Generation College Student -2.86 0.01 -2.03 0.02 -1.83 0.02 
Parent Education Missing -0.94 0.02 0.09*** 0.03 o.05"* 0.03 
Semesters HS English -0.09 0.02 -0.11 0.02 
Semesters HS Math 1.41 0.01 1.05 0.01 
Semesters HS Science 0.04 0.01 -0.06 0.01 
Semesters HS Social Studies -0.64 0.01 -0.65 0.01 
Semesters HS Foreign Language -0.20 0.01 -0.08 0.01 
Semesters HS Arts 0.26 0.01 0.24 0.01 
Honors/AP English 1.38 0.01 1.13 0.01 
Honors/AP Math 1.14 0.01 0.63 0.01 
Honors/AP Social Studies 1.18 0.01 1.19 0.01 
Honors/AP Science 0.86 0.01 0.66 0.01 
Honors/AP Foreign Language 1.29 0.01 1.23 0.01 
Aspirations: Bachelor's 1.48 0.05 0.90 0.05 
Aspirations: 1-2 Years Graduate Study 3.20 0.05 2.35 0.05 
Aspirations: Doctorate 3.02 0.05 2.00 0.05 
Aspirations: Other 1.27 0.07 0.86 0.07 
Aspirations: Missing 3.35 0.06 2.63 0.06 
HSGPA 3.75 0.01 
R-Square 0.06 0.21 0.30 0.35 


Note. All estimates are significant at p < 0.0001 except where ns (p > 0.05) or asterisked. * p < 0.05. ** p < 0.01. *** p < 0.001. 
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Table 16. Regression Models Predicting ACT Math Score 


- Model 1 Model 2 1 ole F=1 I] Model 4 
Variable 
Beta SE Beta SE Beta SE Beta SE 

Intercept 21.15 0.00 22.17 0.00 16.12 0.07 7.70 0.08 
EL, with Supports -5.34 0.04 -4.80 0.04 -1.70 0.13 -1.52 0.13 
EL, no Supports -3.41 0.01 2.31 0.01 1.44 0.02 -1.33 0.02 
Black -3.50 0.01 -3.35 0.01 “2.54 0.01 
American Indian -3.10 0.04 -2.45 0.06 -2.04 0.06 
Hispanic 1.31 0.01 -140 0.01 -0.98 0.01 
Asian 4.02 0.01 2.25 0.02 2.30 0.02 
Pacific Islander -1.75 0.06 -1.03 0.09 -0.78 0.09 
Multiple Race/Ethnicities -0.46 0.02 -0.32 0.02 -0.07** 0.02 
Missing Race/Ethnicity 0.06 0.02 0.33 0.02 0.47 0.02 
Low Income (< $36,000) -1.81 0.01 -1.28 0.01 AN 0.01 
Income Missing 0.36 0.01 0.55 0.01 0.48 0.01 
First Generation College Student -2.10 0.01 -1.33 0.01 -1.17 0.01 
Parent Education Missing -0.69 0.01 0.12 0.02 0.08*** 0.02 
Semesters HS English -0.69 0.01 -0.69 0.01 
Semesters HS Math 2.04 0.01 1.76 0.01 
Semesters HS Science 0.27 0.01 0.19 0.01 
Semesters HS Social Studies -0.74 0.01 -0.76 0.01 
Semesters HS Foreign Language -0.26 0.01 -0.17 0.01 
Semesters HS Arts -0.27 0.01 -0.28 0.01 
Honors/AP English -0.24 0.01 -0.44 0.01 
Honors/AP Math 2.35 0.01 1:95 0.01 
Honors/AP Social Studies 0.54 0.01 0.56 0.01 
Honors/AP Science 0.94 0.01 0.78 0.01 
Honors/AP Foreign Language 0.80 0.01 0.76 0.01 
Aspirations: Bachelor's Ai 0.04 0.65 0.04 
Aspirations: 1-2 Years Graduate Study 2.40 0.04 1.71 0.04 
Aspirations: Doctorate 2.08 0.04 1.27 0.04 
Aspirations: Other 0.91 0.06 0.58 0.06 
Aspirations: Missing 2.44 0.05 1.87 0.05 
HSGPA 3.00 0.01 
R-Square 0.04 0.21 0.39 0.43 


Note. All estimates are significant at p < 0.0001 except where ns (p > 0.05) or asterisked. * p < 0.05. ** p < 0.01. *** p < 0.001. 
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Table 17. Regression Models Predicting ACT Reading Score 


Variable Model 1 Model 2 WY ofe f=1 I] Model 4 
Beta SE |= T=) #-) SE Beta SE Beta SE 

Intercept 22.34 0.00 23.65 0.01 16.87 0.09 744 0.10 
EL, with Supports -8.08 0.05 -7.12 0.05 -3.87 0.17 *3.72 0.18 
EL, no Supports -4.79 0.01 -3.59 0.01 242 0.02 -2.30 0.02 
Black -3.98 0.01 -3.75 0.02 -2.83 0.02 
American Indian -3.58 0.04 -2.72 0.08 -2.27 0.08 
Hispanic -1.55 0.01 -1.69 0.02 1.24 0.02 
Asian 2:22 0.02 0.73 0.02 0.78 0.02 
Pacific Islander -2.46 0.08 -1.76 0.11 -1.47 0.12 
Multiple Race/Ethnicities -0.28 0.02 -0.21 0.02 0.06* 0.03 
Missing Race/Ethnicity -0.27 0.02 0.06* 0.03 0.19 0.03 
Low Income (< $36,000) -1.74 0.01 -1.25 0.02 -1.07 0.02 
Income Missing 0.34 0.01 0.55 0.01 0.48 0.01 
First Generation College Student -2.47 0.01 -1.66 0.02 -149 0.02 
Parent Education Missing -0.83 0.02 0.09** 0.03 0.04"5 0.03 
Semesters HS English -0.02"5 0.02 -0.03"5 0.02 
Semesters HS Math 1.08 0.01 0.76 0.01 
Semesters HS Science 0.01"S 0.01 -0.08 0.01 
Semesters HS Social Studies -0.42 0.01 -0.43 0.01 
Semesters HS Foreign Language -0.14 0.01 -0.04** 0.01 
Semesters HS Arts 0.13 0.01 0.12 0.01 
Honors/AP English 1.40 0.01 1.17 0.01 
Honors/AP Math 0.85 0.01 0.39 0.01 
Honors/AP Social Studies 1.36 0.01 1.37 0.01 
Honors/AP Science 0.83 0.01 0.65 0.01 
Honors/AP Foreign Language 0.89 0.01 0.85 0.01 
Aspirations: Bachelor's 127 0.05 0.76 0.05 
Aspirations: 1-2 Years Graduate Study 2.78 0.05 2.03 0.05 
Aspirations: Doctorate 2.74 0.05 1.84 0.05 
Aspirations: Other 1.10 0.07 0.76 0.08 
Aspirations: Missing 2.89 0.06 2.23 0.07 
HSGPA 3.35 0.01 
R-Square 0.06 0.17 0.23 0.27 


Note. All estimates are significant at p < 0.0001 except where ns (p > 0.05) or asterisked. * p < 0.05. ** p < 0.01. *** p < 0.001. 
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Table 18. Regression Models Predicting ACT Science Score 


y Model 1 Model 2 WY ele F=1 Bi] Model 4 
Variable 
Beta SE Beta SE Beta SE Beta SE 

Intercept 21.53 0.00 22.73 0.00 V727 0.07 9.57 0.08 
EL, with Supports -6.48 0.04 -5.61 0.04 -2.90 0.13 -2.70 0.14 
EL, no Supports -3.70 0.01 -2.62 0.01 -1.61 0.02 1.52 0.02 
Black -3.67 0.01 -3.39 0.01 -2.64 0.02 
American Indian -3.10 0.04 -2.27 0.06 -1.91 0.06 
Hispanic -1.52 0.01 -1.56 0.01 1.18 0.01 
Asian 2.58 0.01 1.10 0.02 1.15 0.02 
Pacific Islander -2.18 0.06 -144 0.09 -1.23 0.09 
Multiple Race/Ethnicities -0.47 0.02 -0.34 0.02 -0.12 0.02 
Missing Race/Ethnicity -0.34 0.02 0.00" 0.02 0.11 0.02 
Low Income (< $36,000) -1.67 0.01 114 0.01 -0.99 0.01 
Income Missing 0.23 0.01 0.44 0.01 0.39 0.01 
First Generation College Student -2.06 0.01 -1.29 0.01 -1.15 0.01 
Parent Education Missing -0.73 0.01 0.10 0.02 0.08* 0.02 
Semesters HS English -0.44 0.01 -0.45 0.01 
Semesters HS Math 1.46 0.01 1.19 0.01 
Semesters HS Science 0.23 0.01 0.16 0.01 
Semesters HS Social Studies -0.55 0.01 -0.57 0.01 
Semesters HS Foreign Language -0.23 0.01 -0.15 0.01 
Semesters HS Arts -0.16 0.01 -0.17 0.01 
Honors/AP English 0.20 0.01 0.01"S 0.01 
Honors/AP Math 1.45 0.01 1.06 0.01 
Honors/AP Social Studies 0.70 0.01 0.71 0.01 
Honors/AP Science 1.04 0.01 0.90 0.01 
Honors/AP Foreign Language 0.67 0.01 0.63 0.01 
Aspirations: Bachelor's 1.10 0.04 0.68 0.04 
Aspirations: 1-2 Years Graduate Study 2.23 0.04 1.60 0.04 
Aspirations: Doctorate 2.18 0.04 1.43 0.04 
Aspirations: Other 0.81 0.06 0.52 0.06 
Aspirations: Missing 2.38 0.05 1.87 0.05 
HSGPA 2.76 0.01 
R-Square 0.05 0.20 0.29 0.34 


Note. All estimates are significant at p < 0.0001 except where ns (p > 0.05) or asterisked. * p < 0.05. ** p < 0.01. *** p < 0.001. 


Difference in ACT Scores 
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Figure 3. Unadjusted and Adjusted Mean ACT Score Differences by Test Group 
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Discussion 


The purpose of this study was to investigate the performance of English Learners 
(ELs) taking the ACT, particularly the scores of ELs who tested with the testing 
supports that ACT began offering in the fall of 2017. Results of this study confirmed 
that the scores of ELs were substantially lower than those of non-ELs, particularly ELs 
who tested with supports. ELs who tested without supports appeared to have higher 
English proficiency than those who tested with supports, as evidenced by their 
higher ACT English scores. Due to the dramatic differences in both English scores and 
overall distributions of Composite scores, as well as differences in demographics and 
high school experiences, it appears that these are two separate populations of 
students. Because this study relied on operational data rather than random 
assignment, it is likely that there were systematic differences between ELs who 
tested with supports and ELs who tested without supports. Similar patterns of 
performance have been found in other studies comparing ELs testing with and 
without accommodations using intact groups (Abedi, Leon, & Mirocha, 2003; Kong, 
Powers, Starr, & Williams, 2012; Young, Cho, Ling, Cline, Steinberg, & Stone, 2008). 
Kong et. al (2012) found that ELs who tested with accommodations had fewer years 
of education in the US and lower English proficiency levels than ELs who tested 
without accommodations. 


ELs in both groups tended to have higher ACT scores in math and science relative to 
English and reading, whereas non-ELs tended to have lower scores in math and 
science relative to English and reading. This is consistent with previous research. 
Several studies have found that ELs score lower than non-ELs across subject areas, 
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Science 
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but the score gap tends to be narrower on assessments that are less linguistically 
complex (Abedi, 2001; Abedi, Leon, & Mirocha, 2003). The ACT English and reading 
tests are completely text-based, whereas the math and science tests contain text as 
well as equations, figures, tables, and other illustrations. 


An additional focus of this paper was the demographic makeup of students across 
test groups, how their high school experiences and HSGPA may differ from those of 
non-ELs, and the impact of these factors on students’ predicted ACT test scores. We 
indeed found that ELs, particularly ELs who tested with supports, were more 
demographically diverse than non-ELs: They were more likely to be non-White, low 
income, and first-generation college students. In terms of high school experiences, 
ELs who tested with supports were less likely to complete a core curriculum 
consisting of four years of English and three years each of math, social studies, and 
science, despite having similar postsecondary educational aspirations as their peers. 
In contrast, ELs who tested without supports largely had similar coursework taking 
patterns as non-ELs. ELs in both groups, particularly those who tested with supports, 
were also much less likely to have taken honors or AP coursework. ELs who tested 
with supports had lower HSGPAs than ELs who tested without supports, and non-ELs 
had the highest HSGPA. All of these factors are related to expected performance on 
the ACT, such that we would expect lower performance from students who take 
fewer core academic courses and have lower grades, regardless of their level of 
English proficiency. 


ELs who tested with supports tended to have lower correlations between their ACT 
scores and HSGPAs than the two other groups. This can be explained in part by the 
lower variability in the ACT scores of ELs, especially those who tested with supports. 
ELs who tested with supports also had more variable correlations between ACT 
scores and HSGPA, with higher correlations in math and lower correlations in 
reading, whereas the other two test groups showed similar profiles but had less 
variability in correlations across subject areas. Students who have higher HSGPAs 
tend to score higher on the ACT, as both are measures of student academic 
performance. However, HSGPA may also be influenced by teachers’ grading 
standards or other student attributes that are not directly related to academic 
knowledge, such as motivation or effort (ACT, 2020). ELs who tested with supports 
were also found to have taken fewer core academic courses and had lower HSGPAs 
compared to ELs who tested without supports and non-ELs. To the extent that 
student grades and ACT scores measure somewhat different constructs, the 
correlations will be attenuated. Correlations may also be attenuated due to 
measurement error. However, due to the large differences in score distributions 
across the three test groups, the extent to which the lower correlations for ELs who 
tested with supports may be related to measurement error is unclear, and additional 
research is needed to further explore this issue. 
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Regression analyses predicting ACT scores based on test group, demographics, high 
school experiences, and HSGPA showed that these factors played a considerable role 
in predicting the scores of students across the three groups, and that the predicted 
performance gaps for ELs in both groups compared to non-ELs were substantially 
reduced when these factors were taken into account. It is likely that there are many 
other factors that are related to the performance gaps but were unavailable for this 
study, such as educational experiences of ELs prior to entering the US, social 
emotional skills, school quality, or other variables that if captured would further 
reduce the adjusted mean differences found in this study. 


An additional finding from this study was that the proportion of ELs who tested with 
supports was very small (less than 1% of the study sample) compared to the 
proportion of ELs who tested without supports (9.5% of the study sample). While the 
reasons for this are unknown, there are some potential explanations that should be 
further investigated. It is possible that students and educators were unaware of the 
existence of the supports, given that they were introduced relatively recently. There 
could also be language or information barriers hindering students’ ability to request 
the supports, particularly since ELs are more likely to be low income, first-generation 
college students as compared to the general population, and thus may not have the 
same resources available when making decisions about their postsecondary plans. If 
eligible students are not making use of the supports, they may be missing out on an 
opportunity to more accurately demonstrate their true achievement level. 


In summary, as the composition of students in the US changes over time, it is 
important that we measure, document, and understand issues related to 
demographics, resources, and access to rigorous coursework, in order to 
contextualize and interpret students’ academic performance as measured by 
standardized assessments. Performance gaps for traditionally underserved 
populations have been documented across many assessments, and understanding 
these issues of differential access can help educators and policymakers understand, 
at least in part, why these gaps are occurring and suggest some ways in which to 
work towards narrowing them, such as ensuring that all students have access to high 
quality educational experiences and that students who may benefit from additional 
resources or assistance are identified and provided appropriate tools to help them 
succeed. 


As the population of ELs grows, and as we continue to monitor their progress toward 
English proficiency and towards college and career readiness, it is important to 
understand the unique challenges that these students face. As prior research as well 
as the findings from this study have shown, ELs are much more likely to come from 
disadvantaged populations and are less likely to be exposed to a rigorous core 
curriculum or advanced or honors courses (Callahan, 2005; Callahan & Shifrer, 2016; 
Johnson, 2019; Sugarman, 2019). This introduces additional challenges on top of the 
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pressing need of these students to learn how to read, write, listen, and speak in 
English. Exposure to high quality academic content is important as well as becoming 
proficient in English; Callahan (2005) found that academic preparation was a 
stronger predictor of ELs’ academic performance than English proficiency. Schools 
should not only focus on English language instruction but also on ensuring high 
expectations and access to rigorous academic content for ELs. To the extent that ELs 
may be receiving English language instruction while their English-proficient peers are 
learning academic content, ELs may fall behind academically compared to their 
peers. ELs face many barriers in their journey towards achieving college and career 
readiness, and the more we can understand about these challenges, the better 
prepared educators and policymakers can be to ensure that this population is 
receiving the support and instruction they need to be successful. 


Limitations and Directions for Future Research 


There are several limitations to this study. One limitation was that scores on 
assessments explicitly designed to measure the English language proficiency of ELs 
were unavailable. While the ACT English test does measure English proficiency, as a 
college readiness assessment, it may not be as sensitive for measuring English 
proficiency of ELs at very low levels and does not measure proficiency in the four 
domains typically measured by English language proficiency tests (reading, writing, 
listening, and speaking). ELs who tested without supports likely had a higher level of 
English proficiency than the ELs who tested with supports, which was supported by 
their higher scores in English on the ACT. It is also likely that both of the EL test 
groups were comprised of students with a wide range of English proficiency across 
the four domains of reading, writing, listening, and speaking, which was not 
examined in this study. 


Most of the covariates used in this study were self-reported. HSGPA was self-reported, 
although previous ACT research has shown that it is highly accurate (and students are 
likely to be motivated to provide accurate responses as the instructions for this 
section of the ACT registration process indicate that “the information you give may be 
verified by college personnel”). The EL status indicator is also self-report, so there may 
be some students who were misclassified. Students were also excluded from the 
study due to missing responses and a “prefer not to respond” option, which may have 
had an impact on the representativeness of the remaining sample. 


There was a large amount of missing registration data for ELs who tested with 
supports. While the reason for this is unknown, it is plausible that limited English 
proficiency could be interfering with their ability to understand and answer the 
questions being asked or slowing down the registration process to the extent that it 
becomes a burden and they begin to skip items. While missing data was addressed 
to some extent by comparing multiple modelling approaches and finding little 
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difference in the results, there is still a possibility of bias in the results. Future research 
is needed to replicate this research with other samples of students and to investigate 
why ELs who tested with supports had a disproportionate amount of missing 
responses. 


ELs who tested with supports were identified by their records in ACT’s 
accommodations system, which contains information entered by school officials on 
students’ behalf. It is possible that there were inaccuracies in the database, meaning 
that there could be students who tested with testing supports but were erroneously 
included in one of the other two test groups. It is expected that these types of errors 
were infrequent and would have had a negligible impact on the findings. 


While students were approved to use testing supports on the ACT, we do not have 
information about whether the supports were actually used. For example, a student 
might have been approved for the use of a word-to-word bilingual dictionary but 
might have forgotten to take it to the test center or brought it but did not use it. In 
addition, we do not know what types of supports students received in the classroom, 
the extent to which they attended mainstream or pull-out courses, or the rigor of 
courses taken aside from a general measure of whether they took honors or AP 
courses, all of which might impact their educational experiences and development 
as well as their HSGPA. If such information was available, we might have a better 
understanding of the differences among the test groups, which may account for 
some of the remaining test performance differences found in the regression analyses. 


This study provided an overview of ELs taking the ACT test with and without testing 
supports, including their academic performance, demographic characteristics, and 
educational experiences. Additional research is underway examining the impact of 
providing testing supports to ELs on their ACT scores. Future research will continue to 
investigate and document reliability, validity, and fairness of scores, including score 
gains of students who first tested without the supports and retested with supports. 
Additionally, a predictive validity study is also being planned to evaluate the 
relationship between ACT scores and college performance for ELs testing with 
supports and how those results compare to findings for non-ELs. Ultimately, the goal 
of the supports is to provide a more equitable testing experience for ELs taking the 
ACT, and additional research is needed to document the extent to which this goal is 
being achieved. 


Notes 


1. Students who test as part of State and District testing can receive 
accommodations or testing supports beyond what ACT approves, but the scores 
resulting from testing under non-standard conditions are not college reportable 
and were excluded from this study. 
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2. The “EL with Supports” test group included students who tested with supports 
regardless of whether they answered the question about receiving EL services, 
whereas the other two test groups required a response to the EL services question. 
This decision was made to maximize the sample size of the “EL with Supports” 
group, as a more stringent inclusion criteria would have substantially reduced the 
sample size. Results were compared across the two inclusion criteria and the 
findings were nearly identical; therefore, the larger sample was retained. 

3. Skewness (jfi,) is a measure of the symmetry of a distribution of data. Skewness of 
zero indicates a symmetric distribution, skewness above zero indicates positive 
skew (i.e., More observations are found at lower values and fewer observations are 
found at higher values), and skewness below zero indicates negative skew (i.e., 
more observations are found at higher values and fewer observations are found at 
lower values). 

4. The percentages of students who were missing registration information was 
compared for ELs who tested with supports conditioned on whether they 
answered the question about receiving EL services. While excluding students who 
did not answer the EL services question would have lowered the percentages of 
missing data somewhat (and substantially decreased the sample size), the ELs 
who tested with supports group still had disproportionately higher percentages of 
missing data compared to the other two test groups. 

5. Correlations between ACT scores and subject-specific HSPGA can be found in the 
Appendix, Table A3. Similar patterns were found regardless of whether ACT scores 
were correlated with subject-specific HSGPA or overall HSGPA. 

6. Extracurricular activities were considered but not included in the models because 
their inclusion added a large number of variables to the models while having very 
little impact on the parameter estimates of other variables or the proportion of 
variance explained. 


ACT Research & Policy | Working Paper | July 2021 3] 


References 

Abedi, J. (2001). Assessment and accommodations for English language learners: 
Issues and recommendations (CRESST, Policy Brief 4). Los Angeles, CA: 
University of California, National Center for Research on Evaluation, Standards, 
and Student Testing. 

Abedi, J. (2002). Standardized achievement tests and English language learners: 


Psychometric issues. Educational Assessment, 8(3), 231-257. 


Abedi, J. (2014). Adapting performance assessments for English language learners. In 
L. Darling-Hammond & F. Adamson (Eds.), Beyond the bubble test: How 
performance assessments support 21st century learning (pp. 185-205). San 


Francisco, CA: Jossey-Bass. 


Abedi, J., Leon, S., & Mirocha, J. (2003). /mpact of student language background on 
content-based performance: Analyses of extant data (CSE Tech. Rep. No. 603). 


Los Angeles, CA: University of California, National Center for Research on 


Evaluation, Standards, and Student Testing. 


ACT. (2006). Benefits of a high school core curriculum. lowa City, IA: ACT. Retrieved 
from 


https:/Awww.act.org/content/dam/act/unsecured/documents/core_curriculum. 
pdf 


ACT. (2012). Enrollment management trends report 2012: A snapshot of the 207] ACT- 


tested high school graduates. lowa City, IA: ACT. Retrieved from 


https:/Awww.act.org/content/dam/act/unsecured/documents/EMTrendsReport 
2012.pdf. 


ACT. (2019a). The ACT. ACT Knowledge Hub. 
https://success.act.org/s/topic/OTOIBOOOOOOUBWpWAO/the-act 


ACT. (2019b). The condition of college & career readiness 2019: National. lowa City, IA: 
ACT. Retrieved from 


ACT Research & Policy | Working Paper | July 2021 32 


http:/Awww.act.org/content/dam/act/unsecured/documents/cccr- 
2019/National-CCCR-2019.pdf. 

ACT. (2020). The ACT technical manual (Version 2020.1). lowa City, IA: ACT. Retrieved 
from 
http:/Awww.act.org/content/dam/act/unsecured/documents/ACT_Technical_M 
anual.pdf. 

ACT. (2021). National norms for ACT test scores reported during the 2020-2021 
reporting year. Retrieved from 
https://www.act.org/content/dam/act/unsecured/documents/MultipleChoiceSt 
emComposite.pdf 

Allen, J. (2015). Anticipated changes in ACT scores and participation rates with ACT 
statewide adoption. lowa City, IA: ACT. Retrieved from 
https:/Wwww.act.org/content/dam/act/unsecured/documents/Statewide- 


Adoption.pdf. 


Callahan, R. M. (2005). Tracking and high school English learners: limiting opportunity 


to learn. American Educational Research Journal, 42(2), 305-328. 


Callahan, R. M., & Shifrer, D. (2016). Equitable access for secondary English learner 


students: Course taking as evidence of EL program effectiveness. Educational 


Administration Quarterly, 52(3), 463-496. DOI: 10.1177/0013161X16648190 


Hakuta, K., Butler, Y., & Witt, D. (2000). How long does it take English learners to 
attain proficiency? (Policy Report 2000-1). Berkley, CA: University of California: 
Linguistic Minority Research Institute. 

Heilig, J. V., & Darling-Hammond, L. (2008). Accountability Texas-style: The progress 
and learning of urban minority students in a high-stakes testing context. 


Educational Evaluation and Policy Analysis, 30(2), 75-110. 


Herman, J.L., & Abedi, J. (2004). Issues in assessing English language learners’ 


opportunity to learn mathematics (CSE Report 633). Los Angeles, CA: 


ACT Research & Policy | Working Paper | July 2021 33 


University of California, National Center for Research on Evaluation, Standards, 
and Student Testing. 

Johnson, A. (2019). A matter of time: Variations in high school course-taking by years- 
as-EL subgroup. Educational Evaluation and Policy Analysis, 41(4), 461-482. 
DOI: 10.3102/0162373719867087. 


Kong, J., Powers, S., Starr, L., & Williams, N. (2012, April 13-17). Connecting English 
language learning and academic performance: A prediction study. Presented 
at the American Educational Research Association Annual Meeting, 
Vancouver, British Columbia, Canada. 

NCES. (2020). The condition of education: English language learners in public 
schools. Washington, DC: NCES. Retrieved from 


https://nces.ed.gov/programs/coe/indicator_cgf.asp. 


Sugarman, J. (2019). The unintended consequences for English learners of using the 
four-year graduation rate for school accountability. Washington, DC: MPI 


National Center on Immigrant Integration Policy. 


Sanchez, E., & Buddin, R. (2015). How accurate are self-reported high school courses, 
course grades, and grade point average? lowa City, IA: ACT. Retrieved from 


http://www.act.org/content/dam/act/unsecured/documents/WP-2015-03.pdf 


US Department of Education. (2016). Non-regulatory guidance: English learners and 
Title Ill of the Elementary and Secondary Education Act (ESEA), as amended 
by the Every Student Succeeds Act (ESSA), Appendix A. Washington, DC: US 
Department of Education, Office of Elementary and Secondary Education. 
Retrieved from 


https:/\www2.ed.gov/policy/elsec/leg/essa/essatitleiiiguidenglishlearners92016. 
pdf. 


Young, J. W., Cho, Y., Ling, G., Cline, F., Steinberg, J., & Stone, E. (2008). Validity and 
fairness of state standards-based assessments for English language learners. 


Educational Assessment, 13(2-3), 170-192. 


ACT Research & Policy | Working Paper | July 2021 34 


Appendix 


Table Al. Number and Types of EL Supports Approved for EL with Supports Test Group 


Count Percent 


Number of Supports 
1 1,714 Nl 


2 3,965 26 
3 4,909 32 
4 4,549 30 
Type of Support 
Translated Test Instructions 6,853 45 
Word to Word Bilingual Dictionary 12,073 80 
Extra Time 14,974 99 
Small Group Testing 8.691 57 


Note. Types of supports do not sum to 100% because most ELs testing with supports were 
approved for more than one support. 


Table A2. Percentages of Missing Registration Category Responses by Test Group and Test 
Type 


EL with Supports EL, No Supports Coy ate =i Ma CoB 8] 0) of e) a uy 
Registration Category Ela Ceyare]| State! National arate! National states 
District District District 
Race/Ethnicity 0.1 19 0.0 5 0.0 5 
Parent Income 62 71 27 15 25 14 
Parent Education 42 67 10 4 8 3 
Coursework/Grades 41 67 14 18 14 14 
Semesters of Coursework 2 54 1 5 1 4 
Honors/AP 85 74 54 19 35 16 
Extracurricular Activities 72 73 27 15 19 13 
Degree Aspirations 46 65 10 5 7 3 


Table A3. Correlations between ACT Subject Scores and Subject-Specific HSGPA by Test 
Group 


7 Xoq 0] 0} (-toa Ug -¥- | 


English Math =t-¥-To [Tale | Science Composite 
EL with Supports 0.25 0.38 0.23 0.26 0.37 
EL, No Supports 0.41 0.44 0.36 0.39 0.50 
Non-EL, No Supports 0.44 0.51 0.38 0.44 0.54 


Note. All correlations were significant at p < 0.0001. 
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